Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tenant Index #834

Merged
merged 39 commits into from
Aug 11, 2021
Merged

Tenant Index #834

merged 39 commits into from
Aug 11, 2021

Conversation

joe-elliott
Copy link
Member

@joe-elliott joe-elliott commented Jul 23, 2021 •
edited
Loading

What this PR does:
Adds support for a tenant index with the following new parameters:

storage:
  trace:
    blocklist_poll_fallback: true
    blocklist_poll_tenant_index_builders: 2

By default all queriers and most compactors will attempt to download a tenant index located at /<tenant id>/index.json.gzip instead of repeatedly listing the bucket. This gzipped json file will contain a list of all metas and compacted metas for this tenant.

blocklist_poll_fallback controls whether or not the various components fall back to scanning the bucket if the index fails.
blocklist_poll_tenant_index_builders is the max number of compactors that attempt to build the index. it could be less depending on sharding.

This PR includes all changes necessary to do the above, as well as related metrics, alerts, and runbook entries. Additionally #825 is fixed because the querier and compactors will now require a polling cycle to complete before they raise their readiness probes.

Which issue(s) this PR fixes:
Fixes #575
Fixes #825

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@joe-elliott
added tenant index and tests
3a126d9 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
note
108a6bc 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Added tenantindex functionality
02e8ec9 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Added poller functionality
2994512 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
fixed poller tests
26e9204 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
EnablePolling
93ad4d2 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
log msgs
97045d3 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
removed incorrect comment
d02a061 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Split out list functionality
6f07a08 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
patched up test
eebb6a7 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
tests
807d97a 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
TestApplyResults
63327bb 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
made tenants return a []string
004a590 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Added TestUpdateSaved
f0eeb2a 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Don't add blocks that are already in the lists
cb22bfa 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
force polling on startup. don't poll querier in single binary
573f2bd 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Added fallback config option
fb6a05b 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Added metrics
16543a5 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
removed tenant from index
90047cf 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
cleanup
354fa4a 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
metrics, alerts, runbook
2ac8014 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Added tenant builders config option
b50ff6b 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
todos
7dd03cc 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
tenant index too old
7545cc8 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
fixed tests
7c6bd57 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
changelog
e70caa9 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
lint
7f32a8e 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Merge branch 'main' into bucket-index
cb4d15a
@joe-elliott
vendor check
adb27ce 
Signed-off-by: Joe Elliott <number101010@gmail.com>
kvrhdn
kvrhdn reviewed Aug 4, 2021
View reviewed changes
Copy link
Member

@kvrhdn kvrhdn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running this locally, I can confirm it fixes #825. The PR in general seems fine, though I'm not super familiar with the compactor.

modules/storage/config.go Show resolved Hide resolved
modules/storage/config.go Show resolved Hide resolved
tempodb/blocklist/list.go Outdated Show resolved Hide resolved
annanay25
annanay25 reviewed Aug 9, 2021
View reviewed changes
Copy link
Contributor

@annanay25 annanay25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very well thought out change. Really like the tenant index creation idea. Added comments.

tempodb/blocklist/poller.go Show resolved Hide resolved
tempodb/blocklist/poller.go Outdated Show resolved Hide resolved
cmd/tempo/app/modules.go Show resolved Hide resolved
mdisibio
mdisibio reviewed Aug 9, 2021
View reviewed changes
docs/tempo/website/operations/_index.md Outdated Show resolved Hide resolved
tempodb/blocklist/poller.go Show resolved Hide resolved
mdisibio
mdisibio reviewed Aug 9, 2021
View reviewed changes
tempodb/blocklist/list.go Outdated Show resolved Hide resolved
@joe-elliott
Fixed e2e tests.
2747fa9 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
PR comments
17867a6 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
docs
8c8c434 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Added testing
ee5db85 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
removed jpe
cfba991 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
doc improvements
05ae732 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Improved tenant index poll tests
ebde24a 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Added fallback testing
566be4d 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Merge branch 'main' into bucket-index
339bb4c
@joe-elliott joe-elliott merged commit 49b130e into grafana:main Aug 11, 2021
@kvrhdn kvrhdn mentioned this pull request Sep 14, 2021
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdisibio mdisibio mdisibio left review comments

@kvrhdn kvrhdn kvrhdn approved these changes

@annanay25 annanay25 annanay25 approved these changes

@achatterjee-grafana achatterjee-grafana Awaiting requested review from achatterjee-grafana

@dgzlopes dgzlopes Awaiting requested review from dgzlopes

@mapno mapno Awaiting requested review from mapno mapno is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@joe-elliott @kvrhdn @annanay25 @mdisibio